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Localisation of humans, objects and robots 
interacting on load-sensing floors 

Mihai Andries, Olivier Simonin, Francois Charpillet 


Abstract —Localisation, tracking and recognition of objects and 
humans are basic tasks that are of high value in applications of 
ambient intelligence. Sensing floors were introduced to address 
these tasks in a non-intrusive way. To recognize the humans 
moving on the floor, they are usually first localized, and then 
a set of gait features are extracted (stride length, cadence, 
pressure profile over a footstep). However, recognition generally 
fails when several people stand or walk together, preventing 
successful tracking. This paper presents a detection, tracking 
and recognition technique which uses objects’ weight. It con¬ 
tinues working even when tracking individual persons becomes 
impossible. Inspired by computer vision, this technique processes 
the floor pressure-image by segmenting the blobs containing 
objects, tracking them, and recognizing their contents through 
a mix of inference and combinatorial search. The result lists 
the probabilities of assignments of known objects to observed 
blobs. The concept was successfully evaluated in daily life activity 
scenarii, involving multi-object tracking and recognition on low 
resolution sensors, crossing of user trajectories, and weight 
ambiguity. This technique can be used to provide a probabilistic 
input for multi-modal object tracking and recognition systems. 

Index Terms —Intelligent systems, Ubiquitous computing, Am¬ 
bient intelligence, Home automation, Force sensors, Sensor ar¬ 
rays, Identification of persons 

I. Introduction 

Ambient intelligence is a domain of research that explores 
how sensing environments can interact with their inhabitants. It 
requires the reconstruction of a model of the environment that 
is used for reasoning. In this context of model reconstruction, 
the localisation, tracking and recognition of objects and human 
beings in the supervised environment become important. Load¬ 
sensing floors were introduced to solve this problem in a non- 
intrusive manner (DEE) The traditional way of recognizing 
humans was by first tracking them, extracting gait features 
and then identify them using clustering techniques 0| or 
Hidden Markov Models (HMM) (T) 0. However, this type of 
recognition failed whenever the extraction of gait features be¬ 
came impossible. This happened when multiple users walked 
alongside, preventing the algorithm from correctly segmenting 
and tracking each of them on the floor. 

This paper presents an object recognition approach which 
localizes and recognizes multiple objects simultaneously by 
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analyzing the load they exert on the floor. As it does not 
extract gait features for recognition, it does not require fine 
tracking of individual persons inside a group. In contrast to 
clustering and HMM techniques, this new approach is based on 
the multiple knapsack problem a, a combinatorial approach 
which uses information about object weight and size. It can 
be used to provide a probabilistic input for a multi-modal 
object recognition and tracking system. This technique was 
implemented in an ambient intelligence setting, where a non- 
intrusive load-sensing floor was used. The main drawback of 
this approach is its computational complexity, due to the sheer 
number of possibilities of correlating known objects to the 
observations made. However, this issue is classically solved 
using dynamic programming. 

For our research experiments, we have designed a sensing 
floor prototype, which allows us to assess how this technology 
can be used for more advanced applications than those avail¬ 
able today on the market. This also allowed us to overcome 
the drawback of the devices available off-the-shelf, which are 
not open and not designed for integrating new sofware. The 
floor has a modular design, being composed of load-sensing 
tiles, whose concept was described in (6). 

The rest of this paper is organised into 6 sections. Section 
[TI| presents the state of art in the domain of load-sensing 
floors. Section [TTT| introduces the load-sensing equipment used 
to experimentally evaluate our algorithm. In section |lVj our 
load-data processing approach is exposed, with an emphasis on 
object detection, tracking, and recognition. Then, experimental 
results for the proposed algorithm are presented and analysed 
in secti on |V| Finally, directions for future work are evoked in 
section IVll 

II. Related work 

The use of floor-sensors in ambient intelligence contexts 
began in the late 1990’s, with projects like the ORL active 
floor (D by Addlesee et al ., the Magic carpet ll24l by Par- 
adiso et al ., and the Smart floor 6) by Orr et al , where they 
provided information for reasoning about the observed space. 
These floors were later integrated into smart environments, 
aimed at delivering assistance services like continuous diag¬ 
nosis of users’ health. These smart environments also inte¬ 
grated assistive robotic technologies with sensing networks. 
Examples include the Gator Tech Smart House made by the 
University of Florida |25), the Aware Home introduced by 
the Georgia Institute of Technology | 26 ] (4), and the Robotic- 
Room system m lf28l developed by the University of Tokyo. 
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Table I: Methods for people identification using floor pressure 


Scientific article 

Year 

Floor sensor 

Major features 

Classifier 

Addlesee et al. [lj 

1997 

strain gauge load cells 

Pressure profile over a footstep 

Hidden Markov Model (HMM) 

Orr et al. (4) 

2000 

strain gauge load cells 

Key points from pressure profile 

K-nearest neighbors (KNN) 

Pirttikangas et al. |[2] 

2003 

electro-mechanical film 
(EMFi) 

Pressure profile over the entire floor during 
walking 

HMM 

Pirttikangas et al. 0 

2003 

EMFi 

Pressure profile over the entire floor during 
walking 

Learning Vector Quantization 
(LVQ) 

Yun et al. 0 

2003 

pressure switch sensors 

Compensated foot centers over 5 consecu¬ 
tive footsteps 

Multi-layer perceptron (MLP) 

Jung et al. |9] 

2003 

pressure mats 

2D trajectories of center of pressure (COP) 

HMM 

Jung et al. (To) 

2004 

pressure mats 

2D positional trajectories of COP 

Hidden Markov Model, Neural 
Network (HMM-NN) 

Suutala and Roning [111 

2004 

EMFi 

Features from spatial, frequency domain 
over a footstep 

Distinction-sensitive Learning Vec¬ 
tor Quantization (DSLVQ) 

Suutala and Roning (12) 

2005 

EMFi 

Features from spatial, frequency domain 
over a footsteps 

MLP, LVQ 

Middleton et al. (13) 

2005 

force-sensing resistor 

(FSR) mats 

Stride length, stride cadence, heel-to-toe ra¬ 
tio 

Not available (N/A) 

Yun et al. fl4ll 

2005 

photo interrupter sensors 

Compensated foot centers and heel-strike 
and toe-off time over 5 consecutive foot¬ 
steps 

MLP 

Yun et al. Q3 

2008 

photo interrupter sensors 

The left footprint pattern and the array of 
sampled transitional footprints over differ¬ 
ent combinations of 2 or 4 footsteps 

MLP 

Suutala and Roning [161 

2008 

EMFi, same as 0 

Pressure and time features extracted from 
pressure profile over a footstep 

MLP, Support Vector Machine 
(SVM) 

Suutala et al. fTTl 

2008 

pressure switch sensors 

Single footstep: length, width, duration, 
number of pixels in the binary map, (min, 
max, mean, std) from the gray-level dura¬ 
tion map; Between footsteps: stride, length, 
stride cadence 

Gaussian Process 

Qian et al. |J8| 

2008 

FSR mats 

2D trajectories of the Center of pressure, 
Pressure profile over time 

Fisher linear discriminant (FLD) 

Vera-Rodriguez et al. Q9J 

2009 

piezoelectric force sensors 

Geometric and holistic footstep data 

SVM 

Qian et al. (20) 

2010 

FSR mats 

Mean pressure, stride length 

FLD 

Vera-Rodriguez et al. 1211 

2010 

piezoelectric sensor mat 

Holistic pressure-time info 

SVM 

Yun et al. (22) 

2011 

photo interrupter sensors 

Foot centers, heel-to-toe time, footprint ge¬ 
ometric data 

MLP 

Vera-Rodriguez et al. l23l 

2013 

piezoelectric sensor mat 

Fusion of time and holistic pressure info 

SVM 

Proposed method 

2015 

strain gauge load cells 

Weight over time 

Knapsack algorithm 


Table [T| lists the floor sensing technologies capable of 
identifying people, updating the lists previously presented 
in (H and d. The current main types of floor pressure 
sensing technologies are: capacitive sensors, piezoelectric sen¬ 
sors, piezoresistive sensors, strain gauge load cells, and photo 
interrupter sensors. 

Being installed inside or under the floor, the load sensors 
perceive only a projection of the forces involved in human 
daily activities. This leaves space for ambiguities in tracking 
and recognition. Thus, whenever floor sensors seemed to be 
insufficient for any of the three tasks (localisation, tracking 
and identification), additional sensors were used in a multi¬ 
modal perspective to solve emerging data ambiguities. Sensing 
floors have been combined with radio-frequency identification 
(RFID) systems l29l . pyroelectric infrared sensors ll30l . wear¬ 
able accelerometers ED ED, audio capture systems m and 
multiple cameras El. 

Load-sensing surfaces are also employed in biomechanical 
and medical laboratories. Examples include the GAITRite gait 
analysis system ED, which is an electronic walkway, and 
the Kistler force plates l36l , which are used for sports and 
performance diagnostics, as well as for gait and balance anal¬ 
ysis. Heller et al. used a force-measuring floor to investigate 
dynamic balance in humans EH, which is correlated to sport 


performance according to previous studies. Rajalingham et al. 
lf38l used in-floor force-sensing to track the 3d body posture 
of pedestrians using Bayesian filters. 

All the presented floors that are capable of human recogni¬ 
tion extract a set of features for their tracking and identification 
task. Addlesee et al. (D recognise humans using their pressure 
profile over a footstep as data, and using Hidden Markov 
Models as classifiers. They also mention the problem of 
interpretation of spread loads, when objects span several tiles 
on a modular floor. Orr et al. m use the vertical ground 
reaction force profile, as well as its derivates. These include the 
maximal load value during heel strike and during toe push-off, 
and the minimal load value recorded during the weight transfer 
from heel to toe. 

Pirttikangas et al. m recognise individual persons walking 
on the floor using the pressure pattern of their gait and 
HMMs. Similarly, Middleton et al. fl3l use the stride length, 
stride cadence, and time-on-toe to time-on-heel ratio and then 
recognise the subjects using a standard distance metric of 
similarity. 

Qian et al. presented an approach to identify people based 
on features extracted from their gait m. They used a large 
area (180 square feet), high resolution (1 sensor per cm 2 ), net¬ 
worked pressure sensing floor which employed force-sensing 
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resistors (39). 

Schmidt et al. 0 queried a database of known objects 
whenever a change was detected in the total weight of a scene, 
to see if there exists an entry that has the same weight as the 
absolute difference in weight detected. However, this process 
was neither probabilistic, nor could it detect simultaneous 
introductions or removals of objects from the scene. 

Morishita et al. in [40) presented a high-resolution floor 
sensor composed of pressure switches, which provided binary 
information about the presence or absence of load on them. Its 
high resolution allowed to obtain sharp images of the surfaces 
in contact with the floor, such as footprints or shoe soles. It was 
suggested that an image processing software could regonize 
footprints. 

Murakita et al. (4l) performed multi-user human tracking 
on the VS-SS-F InfoFloor system using the Markov Chain 
Monte Carlo method. However, the employed floor sensors 
gave only a binary information about the occupation of its 
constituent tiles. Tracking would fail whenever two or more 
targets crossed their paths, generating tracking ambiguity. 
Attempts were made to solve this problem by fusing the 
information from the floor sensors with that from on-body 
acceleration sensors (32). 

Savio et al. [421 identified footsteps on a smart carpet 
with integrated binary capacitive sensors, using clustering 
algorithms based on Maximum Likelihood Estimation and 
Rank Regression analysis. This allowed the extraction of user’s 
trajectory. 

Valtonen et al [43 ] presented a 2D human positioning and 
tracking system, which used a low-frequency electric field to 
locate humans on the floor. The system could only detect 
conductive objects, and did not provide information about the 
weight of objects. 

Similarly, capacitive sensing floor mats capable of detecting 
and tracking objects in the environment were used by Braun 
et al. m. However, as this floor did not measure load forces, 
it could not recognize objects on its surface by their pressure 
profile. 

Lombardi et al. El developed a tiled sensing floor, with 
tiles containing sensing stripes, and which was capable of 
tracking a walking human. They used a Randomized tree clas¬ 
sifier to identify footsteps from the pressure signal. However, 
the floor did not implement any human recognition abilities. 
It is worth mentioning that they treated the floor pressure data 
as if it were an image. 

Shen and Shin (46) developed a floor that uses an optical 
fiber sensor. It employs Brillouin Optical Correlation Domain 
Analysis (BOCDA) to calculate the location and the stress on 
the sensor. The floor was able to track two persons simultane¬ 
ously. However, no identification abilities were developed or 
reported. 

Yun et al. worked on several sensing floor prototypes (8) 
Ifl4l (I5l . Their latest prototype, the UbiFloorll (22) . uses an 
array of tiles equipped with photo interrupter sensors. It uses 
multilayer perceptron networks to identify individuals based 
on the extracted features (stride length, foot angle, heel strike 
time, etc.). 

Vera-Rodriguez et al. described a high-resolution pressure¬ 


sensing floor, which employs piezoelectric sensors mounted 
on a printed circuit board, and placed under a conventional 
mat ED. The authors extracted the ground reaction force of 
footsteps (2l) and their footprints [19), and performed human 
recognition using a Support Vector Machine. Recognition was 
performed using a database of 120 known people, the largest 
database to that date. 

Lee et al. E3 presented a network of force-sensing resistors 
that can track users and allow them to interact with the 
smart environment by tapping the floor, onto which a visual 
interface is projected. Similarly, Visell et al. (48l l used a tiled 
load-sensing floor as a human-computer interface, where an 
image of the interface is overlayed on the floor, on which 
users can press virtual buttons with their feet. The Active 
Gaming company proposed the pressure sensitive Lightspace 
Floor [49], which is an interactive gaming platform combining 
pressure sensors with LEDs for visual feedback. 

More recently, sensing floors products like the Sens- 
Floor m (a floor network of capacitive proximity sensors), 
Capfloor (44) (a network of capacitive sensors), and Floor- 
InMotion ED started being commercialised by companies, 
mainly for the senior care industry. An innovative energy 
harvesting sensing floor has also been proposed in (52) . 

Concerning object tracking on sensing floors, inspiration can 
be sought in the field of computer vision, where techniques 
such as Bayesian Filtering [53], Joint Particle Filtering (54) . 
Probabilistic Multi-Hypothesis Tracking G3, and Joint Prob¬ 
abilistic Data Association Filtering [56] have been applied for 
tracking multiple targets. Challa et al. provided an overview 
of these techniques in Fundamentals of object tracking (57) . 
Suutala et al. (58) used Gaussian Process Joint Particle Fil¬ 
tering to track humans on a tiled floor equipped with binary 
switch sensors. However, their algorithm did not use weight 
information to improve object tracking, as this information 
was not provided by their hardware. In the case of pressure¬ 
sensing floors, we can also exploit the weight information to 
evaluate the generated tracking hypotheses. 


In this paper we present an algorithm that detects, tracks 
and recognizes objects by using only the information about 
their size and weight. It offers a solution to the problem 
of interpretation of spread loads, when objects span several 
tiles on a modular floor. In comparison to the aforementioned 
tracking techniques, which exploit only binary data about the 
presence or absence of objects, our tracking algorithm also 
exploits the weight data provided by load-sensing floors, as 
detailed in Section IV-B As it tracks and recognizes objects 
individually or in groups, it is more fault tolerant as opposed 
to algorithms that extract gait features, which require fine 
segmentation and tracking of targets. This technique can boost 
recognition when used complementarily with algorithms that 
extract features from the human gait, but can also serve as a 
gracefully degraded recognition mode whenever these fail. 
III. Load sensing equipment 


We have implemented our object recognition algorithm 
on the SmartTiles platform f6), which is installed in our 
ambient intelligence prototype apartment (see Fig. [4]). This 
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load-sensing floor is composed of square tiles, each equipped 
with 4 pressure sensors (strain gauge load cells), two ARM 
processors (Cortex m3 and a8), and a wired connection to the 
four neighbouring cells (see Fig. [I]). The processing units were 
manufactured by HikotQ The tiles’ architecture is presented 
in Fig. [2] As shown in the diagram, both centralized and 
decentralized applications can be supported, thanks to the 
computing units embedded in the tiles. The tiles form a 
network of load-sensors, as represented in Fig. [5] 



(a) Underside of a load-sensing 
tile. The load sensors are in the 
corners of the tile. 



(b) The SparkFun SEN-10245 
load sensor used. Source: 
www.sparkfun.com/products/ 
10245 


Figure 1: An image of a tile and a load-sensor. 
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Figure 2: Tile architecture. Low-level firmware is the violet 
block, blue blocks form the middleware, while high-level 
software blocks are in yellow. 


1 http://www.hikob.com/ 


This prototype was originally designed as a medium of 
interaction for robots with distributed control, in an ant-like 
fashion. The computing unit available on each tile can register 
a virtual pheromone trace, that can then be transmitted to other 
robots, using either wired or wireless communication. 

In a different perspective, the sensing-floor acts as a sensor 
for an ambient intelligence. It can measure pressure forces 
with the load sensors installed under the tiles, measuring static 
weights with a precision of up to ± 2 kg. The floor can also 
detect disturbances in the surrounding magnetic field caused 
by the presence of robots, using magnetometers embedded on 
the processing units of the tiles. 

Each tile also has an embedded accelerometer, that allows 
it to detect shocks that can be caused by objects or humans 
falling on the ground. Floor devices of similar functionality, 
such as the SensFloor f50l that can detect people lying on 
the ground are already employed in nursing homes in France. 
Each tile has 16 light-emitting diodes which provide visual 
feedback. 

The floor can localize the exerted punctual pressures, with 
an accuracy beyond the size of a tile. Punctual pressures can 
be located through a calculation of the center of pressures 
measured by the load sensors. The precision is influenced by 
the signal to noise ratio, as visible in Fig. [3] 

Several functionalities have already been implemented on 
this prototype floor, including weight measurement, fall de¬ 
tection, and footstep tracking. The floor’s ability to perform 
high resolution pressure sensing by shifting the objects on the 
sensing tiles has been demonstrated in (59). 

This type of tiled sensing floor also has its inconvenients: 

• it is incapable of distinguishing between objects of equal 
weight; 

• its resolution depends on the size of the tiles, which are 
usually quite large (30 cm x 30 cm or bigger); 

• its sensitivity depends on the sensitivity of its sensors; 

• perception of forces is limited to the floor plane; 

This floor is also capable of detecting and measuring foot¬ 
steps with high accuracy, extracting them using the variations 
in the translation speed of the center of pressure, as described 
in [ 601. We also implemented heuristic real-time multi-user 
localisation (without user identification) in an indoor setting 
using this prototype floor. This paper focuses on the object 
detection, tracking and recognition capability of such a load¬ 
sensing tiled floor. 

IV. Methodology: load data processing flow 

Parallels can be drawn between data processing in the 
context of computer vision and that of load-sensing floors. The 
field of view of a camera is analagous to the surface covered 
by a sensing floor. The light-intensity bitmap image generated 
by a camera is analagous to the load image generated by a 
load-sensing floor. This hints that traditional image processing 
techniques can be employed to solve similar problems in the 
context of load-sensing floors. 

The traditional data processing flow in computer vision usu¬ 
ally consists of the following steps: background subtraction, 
blob detection, blob tracking, and blob recognition. The data 
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flow processing that we propose for load-sensing floors is 
similar, and has the following structure: background subtrac¬ 
tion, blob detection using connected-component labeling , and 
a feedback loop perfoming blob tracking and localisation of 
objects (see Fig. The algorithm receives as input: 

• the force values registered by the sensors composing the 
floor when there is nothing on it (i.e. the zero values used 
for background subtraction); 

• the values of forces recorded at time t, together with the 
coordinates of the load sensors that sensed them; 

• a list containing the models of objects known to the floor. 

The object models have the following structure: object 

name, mass (in kilograms), and length (in meters). 

A. Object detection 

Objects are detected on the floor by background sub¬ 
traction and subsequent connected-component labeling. The 
background subtraction allows to process the data from sen¬ 
sors that perceived force values above zero, filtering out all 
other sensors. Then, connected-component labeling l6TI links 
together all sensors that are potentially supporting the same 
object, thereby forming blobs. It uses the length of the largest 
known object as a proximity threshold: if two sensors detected 
pressures over the noise threshold, and if the distance between 
the sensors is smaller than the size of the biggest known object, 
these are linked together, forming a connected component. 

The size of the largest known object is calculated from 
the list of known object models. After this phase, any object 
present on the floor is guaranteed to be contained by one blob 
at most. On the other hand, a blob may contain one or several 
objects. 

For simplicity reasons, we will consider that there is no oc¬ 
clusion in our system, which occurs when a tile malfunctions 
and stops sending load data. 

In our implementation, the set of sensors is represented as 
a graph (see Fig. Blobs are formed by the sensors left after 
background subtraction, which are linked using connected- 
component labeling. The blobs correspond to the connected 


components present in this graph. Figure [6] shows the set of 
load sensors embedded into the floor, where each sensor is 
represented as a dot. The blobs detected by the floor were 
then overlayed onto this image. 

B. Object tracking 

After the detection of blobs on the floor, we can try to 
infer the objects located in these blobs by using their weight. 
However, the load force detected by the sensors oscillates 
during activities such as walking or squatting and standing 
up (see Fig. [7] for an example). Thus, the value of this force 
cannot be directly converted into an estimation of an object’s 
mass. Nevertheless, the value of this force oscillates around the 
weight of the object or person, as mentioned in [1]. Therefore, 
it is possible to approximate the total weight of objects inside 
a blob, by calculating the blob’s average weight over a sliding 
window of time. This requires blob tracking. 

An adequate solution to this problem is to use a tracking 
technique that takes into consideration the different ways in 
which blobs can evolve. A blob can appear in the scene, 
disappear, remain constant, merge with other blobs, split into 
several blobs, or it can exchange contents with another blob. 
We propose a method that explores the entire search space 
of joint blob evolution hypotheses (except for remote content 
exchange between blobs, rarely encountered in practice), and 
sorts these hypotheses according to a given criterion. Intu¬ 
itively, the optimal solution should minimize the total distance 
travelled by the blobs inside the scene between two instants of 
time, as well as minimize the weight difference between the 
correlated blobs in two neighboring time frames. We define 
penalties for each type of blob evolution, which are used for 
ranking the tracking hypotheses (see Table E 

An appear evolution penalizes the weight of the appeared 
blob, as well as the distance between the new blob and 
the entry/exit location of the environment. Symmetrically, a 
disappear evolution penalizes the weight of the disappeared 
blob, and the distance to the exit point. A split evolution 
penalizes the difference between the weight of the parent blob 


Center of presssure scattering (tile load 7kg) 


Center of presssure scattering (tile load 12kg) 


Center of presssure scattering (tile load 20kg) 



0) 0.1 
Q) 

E 



Width (in meters) 


-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 
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Figure 3: The scattering of the calculated center of pressure, caused by the sensor noise. The thick black square represents 
the load-sensing tile. Scattering is shown for 3 different loads (7kg, 12kg and 20kg) at 16 different locations of the exerted 
punctual pressure, marked by black circles. The higher is the ratio of signal to noise, the less scattering is observed. The 
indicated load does not include the weight of the tile itself, which is 10.7 kg. 
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and the total weight of the child blobs. A merge evolution 
penalizes the differences between the total weight of the parent 
blobs, and the weight of the (unified) child blob. In terms of 
distance, both split and merge penalize the euclidean distance 
between parent and child blobs. However, this distance penalty 
is considered nil for the parent and child blobs that overlap 
and occupy the same surface tiles (e.g. all the split child blobs 
that are contained within the surface of the parent blob; all the 
merged parent blobs that are contained within the surface of 
the unified child blob). 

The final score of a hypothesis is obtained by first dividing 
the distance penalty and the weight penalty by their corre¬ 
sponding average noise values, squaring the results, and then 
summing them up to obtain the mixed final score. The joint 
blob evolution hypothesis with the lowest penalty is considered 
to be the most probable one. 


C. Object recognition 

Object recognition on load sensing surfaces can be per¬ 
formed by using the weight of objects, or by using their 
surface of contact with the floor f59l . Recognition by weight is 
trivial when tracking single objects or when performed on high 
resolution pressure sensors, that can easily segment objects on 
the floor. However, the problem is less trivial when tracking 
multiple entities, each with multiple points of support, and 
which interact on noisy, low-resolution sensors. 

Background subtraction, connected-component labeling, 
and blob tracking, described in the previous sections |IV-A| 
and |IV-B[ reduce the problem to recognizing the contents of 
blobs of known weight, which support the weight of one or 
more objects in their entirety. This allows us to model the 
recognition task as an instance of a Multiple Knapsack Prob¬ 
lem, interpreting the weights of detected blobs as knapsacks’ 
volumes, that have to be optimally filled with known objects’ 
weights. This is based on the hypothesis that the average 
weight of a blob is optimally matched by the weights of the 
objects it contains (see Fig. [8]). 

This can be formalised as follows: 


• O is the set of known objects; 

• V(0) is the set of all combinations of known objects (it 
is the power set of O ); 

• C = { blobi ,..., blob n } is the set of all blobs observed 
at a given time t. Each blob is defined by its location and 
weight. 

All the possible assignments of objects to blobs are con¬ 
sidered and ranked in ascending order, by using the total 


mismatch in weight between the blobs and the objects assigned 
to them (see Fig. [9]). For each hypothetical assignment of 
objects to blobs, these mismatches are squared and then 
summed, so as to give preference to small mismatches, rather 
than large ones. The weight mismatch (or the penalty) of an 
assignment of objects to blobs is given by: 

total blobs 2 

^weight (blob^d) — weight (contents (blob^) (1) 

id= 1 

The size of the search space, that is the number of possible 
assignments to analyse, is given by: 

(number of blobs + l) (number of >“ ow " ob j ects ) (2) 

As highlighted by eq. [2] there is a risk of a combinatorial 
explosion when performing this exhaustive search. This can be 
dealt with using traditional techniques like branch and bound , 
and dynamic pro gamming. 

Given the contents of blobs in the previous timestep, and 
given a hypothesis on how the blobs have evolved inside the 
scene from the previous to the current timestep, we can infer 
the contents of blobs at the current timestep. However, this 
requires bootstrapping the knowledge about the contents of 
blobs at some initial time t sta rt- 

As the blobs inside the scene evolve, the candidate recogni¬ 
tion solutions will cumulate penalties over time. The candidate 
solution with the minimal total penalty over time is considered 
to be the best guess (see Fig. |T0| ). 

The result of the recognition algorithm is a list of as¬ 
signments of objects to blobs, ordered according to their 
cumulated penalties. Intuitively, the assignment having the 
minimal penalty is considered to be the most probable one. 

For probabilistic reasoning algorithms, a measure describing 
the probability for an assignment of not being the correct solu¬ 
tion can be introduced: this is the penalty of the assignment, 
normalized using the sum of all assignments’ penalties (see 

eq- 13)- 

P(-<Assignmentk) = - . - (3) 

Y) Penalty (Assignment ^ d ) 

id= 1 

V. Experiments 

We evaluate our approach by running experiments with 
humans performing daily life activities: doing the morning 
routine (waking up in the bed, using the toilet, having break¬ 
fast), and receiving a visitor (opening the door, leading the 



Appear / Disappear 

Merge 

Split 

Distance penalty 

( distanceTo(entrance/exit) ^ ^ 

V avg COP scattering noise ) 

non-overlapping 

parents 9 

Y^ ( distanceTo(childBlob) \ A 


non-overlapping 

children 

Y^ ( distanceTo(parentBlob) \ ^ 



V average COP scattering noise J 



Z —4 V average COP scattering noise J 


Weight penalty 

T weight(blob) 1 ^ 


parents 

^ 'y ^ weight(blob)^j — weight(childBlob) 

2 


( children \ 

weight (parent Blob)— I ^ ^ weight(blob) J 

2 

L avg pressure noise J 


average pressure noise 



average pressure noise 



Table II: Calculation of penalties for each type of blob evolution 
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Load profile, tile 26 

180 - 

160 - 
140 - 



Time (in seconds) 

Figure 7: The load profile of a person squatting and jumping on 
a load-sensing tile. Notice that the load oscillation is centered 
around the mass of the person, which is 60 kg. 


Figure 4: 3D model of the intelligent apartment prototype, 
with the load-sensing floor. 



Figure 5: 2D image of the tiles composing the floor, with the 
sensors highlighted in red. Gray tiles are not equipped with 
sensors. 



Figure 6: Object recognition sample. The floor load sensors 
are represented as little black dots. The detected blobs are 
colored in green. The numbers in black show the average blob 
weight, calculated over a time window. The red dots show the 
position of blobs’ centers of mass. The text in red shows the 
recognition guess. 



65 kg 30 kg 


Alice (60 kg) 
Chair (5 kg) 
Bookcase (30 kg) 
Bob (90 kg) 



Figure 8: Object recognition modeled as a Multiple Knapsack 
Problem. 


Assignment^ 


Assignmentj 


Assignment( blobs+1 )objects 

Objecti -A 0 


Objecti —> Blob 2 


Objecti —>• Blob k 

Object^ -A 0 


Object 2 -A 0 


Object 2 -a Blob/c 

Object^ -A 0 


Object n -A Blobfc 


Object n -a Blob/j 


Figure 9: All the possible assignments of known objects to 
blobs are evaluated and ordered according to how well the 
blobs are matched in terms of weight by their contents. The 
assignment having the minimal weight mismatch is considered 
as most probable. 



Figure 10: The optimal assignment of objects to blobs cumu¬ 
lates the minimal penalty over time. 
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(a) The bedroom and living room. 



(b) The living room, the bathroom and the kitchen. 

Figure 11: The prototype apartment with the tiled sensing 
floor. 


visitor into the living room, having a chat while seated, 
eating a cake, leading the visitor to the exit). All scenarii 
involve multi-object detection, tracking and localisation. The 
experiments took place in our prototype apartment (Fig. ET 
During the whole duration of these activities, the sensing 
floor had to localize the persons and objects inside the scene. 
The center of pressure (COP) of a blob was considered 
as the location of all the objects contained by this blob. 
An approximation of the ground truth was provided by a 
Qualisys [^] motion tracking system (tracking error below 1 
mm), which recorded the vertical projection of markers placed 
on objects’ approximate centers of mass. Each human had a 
reflector placed on his waist, so that its vertical projection 
onto the ground plane would approximately correspond to 
his COP (fig. |13a| and |14a| ). The experimental results are 
presented as measurements of the localisation precision. These 
measurements were made only when both localisation data 
were available: the approximate ground truth given by the 
motion tracking system, and the localisation provided by the 
floor. This explains the interruptions in the curves showing the 
localisation precision. 


saccadic movements of the COP which are characteristic for 
the human gait. As we had no ground truth for the localisation 
of robot’s center of pressure, we used an approximation using 
the data from the motion tracking system. Considering that 
the robot is rigid, we could estimate the position of its COP 
using the least squares method, by calculating the point which 
minimized the quadratic distance error between itself and the 
center of pressure calculated by the sensing floor. 

The robot was localised by the sensing floor with an average 
precision of 8 cm for free movement, and a standard deviation 


of 5 cm, as shown in Fig. 12 


B. Morning routine scenario 

The morning routine scenario involved a person performing 
a set of daily life activities, such as: sleeping in bed, using 
the toilet, having breakfast, and leaving the house (see Fig. 
13). The challenges of this scenario included tracking multiple 
interacting entities on a low resolution sensor, as well as the 
presence of ambiguity between objects of similar weight. 

As we had no system to provide us with the ground truth for 
the localisation of a person’s COP, the measured localisation 
error obtained using the COP approximation given by the 
motion tracking system is expected to be higher than the 
real localisation error. In contrast to rigid robots, humans are 
flexible. This did not allow us to calculate an approximation 
of the COP using the least squares method, as we had done 
in the baseline case with a rigid robot. 

The localisation errors for the human (average error 13 
cm) and the bed (average error 19 cm) are shown in Fig. 
EH When the person interacts with the bed, the two are 
segmented together in a single blob, with the COP closer to 
the heavier human, which explains his better localisation. The 
localisation error is the biggest at the beginning and end of 
each interaction, when the two entities begin approaching each 
other, forming an elongated blob. The localisation error is at 
its lowest during the close interactions between objects, when 
the blob regrouping the interacting objects is compact. We 
observe 15 cm of localisation error for the bed when it is at 
rest, and a higher error during interactions, which depends on 
the proximity with the interacting entity and its relative weight. 


Fig. 13c shows the localisation errors for lightweight ob¬ 
jects, such as the chair (5.5 kg) and the dish with the breakfast 
(4.9 kg). A heavy plate was chosen to overcome the noise 
threshold of the floor sensor. We observe the same effects as 
previously described: the localisation error increases in the 
proximity of humans, due to their segmentation in a common 
blob, with the COP closer to the human. 


A. Baseline precision 

To gain an understanding of the baseline precision of 
the floor sensor, we performed an experiment with a non- 
holonomic 4-wheeled robot (robuLAB-10 by Robosof0 
rolling on the floor of the apartment. The idea was to track the 
fluid movement of an autonomous robot, as compared to the 

2 http ://www. qualisys .com/ 

3 http://www.robosoft.com/ 


C. Receiving a visitor scenario 

The receiving a visitor scenario involved a person hosting 
someone in his house. The guest would be greeted at the door 
by the host, enter the living room of the apartment, take a seat, 
wait for the host to bring something to eat, have a chat with 
the host, and then leave the house. The challenge was to track 
and locate multiple interacting persons with a low resultion 
sensor. 
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(a) A robuLAB-10 robot navigating on the 
sensing floor. 


trajectories 



y position (meter) 


(b) The robot trajectory (red), and the 
localisation given by the floor (blue). The 
average localisation error is 8 cm, with a 
standard deviation of 5 cm. 


position errors 
0.30,-s-|-s- 


0.25 



all data 1 tile 2 tiles 3 tiles 4 tiles 


(c) Tukey box plot presenting an analysis 
of the floor’s localisation error, depending 
on the number of tiles supporting the robot 
at the time of localisation. 


Figure 12: The robotic navigation scenario 


The localisation results are shown in Fig. |14b| The average 
localisation error for interacting persons is around 20 cm. The 
drop in the localisation precision occurs when the two persons 
walk or stand nearby, occupying a contiguous space in terms 
of tiles, which prevents them from being segmented separately. 
Again, the measured localisation error for humans is expected 
to be higher than the real localisation error, as we could not 
approximate the position of the human’s COP with the least 


squares method, as we did it with a rigid robot in section V-A 


D. Discussion 

The presented algorithm works independently of the floor 
resolution (i.e. density of sensors per m 2 , size of the floor 
tiles). However, the bigger the tiles are (the lesser the floor- 
image resolution is), the coarser the results of the object 
detection algorithm will be. Also, coarser object detection 
results introduce more ambiguity in object recognition and 
localisation. Therefore, it would be interesting to have a 
prototype with a higher sensor density (smaller tiles in our 
case), as well as less noisy sensors. 

The use of these tiles dictates their smallest practical size: 
for footstep tracking applications, tiles having the size of a 
foot are sufficient. For more fine-grained details, as required 
by biometrical applications, other types of sensing floors may 
be more adequate (e.g. pressure mats), if judged by price per 
unit of sensing surface, or by their fabrication complexity. 
It would also be interesting to have sensors that capture the 
xyz components of the ground force. This would allow the 
reconstruction of the human body posture, given a model of 
the human body and of its constraints. 

VI. Conclusion and perspectives 

This paper presents a technique for detecting, tracking 
and recognising objects on load-sensing floors, using objects’ 
weight as discriminative feature. The proposed object seg¬ 
mentation algorithm is a variation of connected-component 
labeling , inspired by the computer vision community, with the 


additional property of having entire objects segmented into 
blobs. This allows the interpretation of spread loads, when 
objects span several tiles on a modular floor. The proposed 
tracking algorithm considers the different ways in which blobs 
can interact, identifying the most probable hypotheses for 
the way the blobs have evolved between two timesteps. This 
allows to infer the objects contained in the segmented blobs, 
given their contents at the previous timestep, and given a 
hypothesis on the evolution of blobs. The resulting possible 
assignments of objects to blobs are ranked by the mismatch 
between the weight of blobs and of objects assigned to them. 
This is reminiscent of the multiple knapsack problem , with 
blobs acting as containers that have to be optimally filled 
with known objects, identifying the optimal solution using 
Least Squares. The whole localisation algorithm was evaluated 
in experiments with humans performing daily life activities: 
executing the morning routine, and receiving a visitor. Chal¬ 
lenges included the segmentation, tracking and recognition 
of multiple interacting entities using a low resolution sensor, 
as well as disambiguation between combinations of objects 
of similar weight. The average error for human localisation 
was approximately 20 cm. The result of this algorithm can 
be modelled as a probability distribution over all possible 
assignments of objects to the blobs detected on the floor. 
This allows for easy integration of this algorithm into a 
multi-modal object recognition architecture. This technique 
can boost recognition when used complementarily with algo¬ 
rithms that extract features from gait, but can also serve as a 
gracefully degraded recognition mode whenever these fail. 

Future work will include fine-grained tracking, obtained by 
assigning each detected object to a separate layer. This should 
allow to continuously update the set of objects composing the 
background, and would consequently improve segmentation. 
We are also working on labeling the interactions between 
humans and objects, which can be roughly observed using 
this technique. We also plan to use this data to generate logs 
detailing the activities performed by a person during the day: 
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(a) An image from the Morning routine scenario. 


Localisation error: Morning routine scenario 



Time (in seconds) 



(a) An image from the scenario Receiving a visitor. 


Localisation error: Visitor scenario 

Host localisation error - 

Guest localisation error - 

Dish localisation error- 





Time (in seconds) 


(b) Localisation error for the Morning routine scenario. During 
each interaction between the person and the bed, the center 
of pressure is located between the interacting entities, closer 
to the heaviest one (the human, in this case). The spikes 
between 10-30s, and between 62-68s are caused by the human 
approaching and leaving the bed during the interactions. 


(b) Localisation error for the scenario with a person hosting 
a visitor in his house. The drop in the localisation precision 
occurs when the two persons walk or stand nearby, occupying 
a contiguous space in terms of tiles, which prevents them from 
being segmented separately. Average human localisation error: 
20 cm. 



o y - m - warns hb - 

0 10 20 30 40 50 60 70 80 


Time (in seconds) 

(c) Localisation error for the lightweight objects in the Morn¬ 
ing routine scenario. Occasional, short-time errors in the 
correct assignment of objects to blobs generate the thin spikes 
in the localisation error. These are due to ambiguities between 
objects of similar weight. The spikes in the chair localisation 
error are due to the human proximity, having as effect the 
segmentation of the two in a single blob. 

Figure 13: The Morning routine scenario 


Figure 14: The Visitor scenario 

how many times a person got out of bed, how many steps did 
he make, how many persons are there in the room, etc. These 
activity plots are useful in hospitals and retirement homes, as 
they allow to trace the overall health state of a patient. 
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